Logical analysis of data: classification with justification

Learning from examples is a frequently arising challenge, with a large number of algorithmsproposed in the classification and data mining literature. The evaluation of the quality ofsuch algorithms is usually carried out ex post, on an experimental basis: their performanceis measured either by cross validation on benchmark data sets, or by clinical trials. None ofthese approaches evaluates directly the learning process ex ante, on its own merits. In thispaper, we discuss a property of rule-based classifiers which we call “justifiability”, and whichfocuses on the type of information extracted from the given training set in order to classifynew observations. We investigate some interesting mathematical properties of justifiableclassifiers. In particular, we establish the existence of justifiable classifiers, and we show thatseveral well-known learning approaches, such as decision trees or nearest neighbor basedmethods, automatically provide justifiable classifiers. We also identify maximal subsets ofobservations which must be classified in the same way by every justifiable classifiers. Finally,we illustrate by a numerical example that using classifiers based on “most justifiable” rulesdoes not seem to lead to overfitting, even though it involves an element of optimization.